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ABSTRACT 


This thesis investigates the feasibility of applying a knowledge-based 
approach to the problem of estimating the future success of a major defense 
acquisition. This thesis will model the US Government Accountability Office (US 
GAO) knowledge based methodology for evaluating programs. This methodology 
relies on three sets of knowledge, namely, technology, design, and production. In 
particular, the technology dimension is measured by Technology Readiness 
Level (TRL), as defined by NASA. In addition, the methodology relies on 
assessing the design readiness of a program by examining the status of the 
release of engineering drawings. Finally, the US GAO methodology assesses 
the production readiness of a program by examining the status of Statistical 
Process Quality Control (SPQC) procedures. This thesis also presents to the 
ROKAF decision maker an independent, knowledge-based estimation of the 
ROKAF T-50 program by applying the US GAO methodology to the T-50 


program. 
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EXECUTIVE SUMMARY 


This thesis investigates the knowledge-based approach developed by the 
US GAO to estimate if the weapon acquisition program could meet requirements 
without cost overruns and schedule delays. To manage the major weapon 
acquisition program and achieve the key results, the US GAO has developed 
three knowledge points; technology maturity, design stability and controlled 
production stability. The T-50 acquisition program, the newly developed training 
aircraft of ROK Air Force, is subjected to an assessment by the GAO 
methodology to ascertain if the program would enjoy good progress in the 
research and development phase. The 10 US programs’ data are also compared 
with the T-50 program to evaluate the technology maturity of those programs and 
judge if they are following GAO guidance. 


The following shows the status of whether each program in our dataset 
follows the GAO guidance or not. GAO considers each of these as a “best 
practice” in the major weapon acquisition process 

° Technology Readiness Level(TRL) above 7 


° 90 percent achievement of engineering drawings at Critical Design 
Review (CDR) 


e Production supported by Statistical Process Control (SPC) 
The GAO knowledge based-methodology about T-50 program finds some 
conclusions with the above rules. 


° Since the T-50 program achieved a TRL of 6.3 at the development 
stage, we believe that the T-50 program will achieve the goal of 
TRL 7 with the trend of technology development. 


e The releasable drawings of T-50 program at CDR is not at a 
sufficient level to ensure design stability. T-50 has more design 
drawings than the advanced programs in our dataset as well as the 
unmanned programs in our dataset. The T-50 program’s 
achievement of design drawing, 67%, at CDR is below the “best 
practice” guidance of the GAO. 


XV 


T-50 prototype production had been performed without fully 
developed statistical control and production is now at the beginning 
phase of mass production. However, there has not been an 
instance where SPC has been implemented directly for each 
component of the T-50 prototype model. On the other hand, the T- 
50 program has a specific plan of SPC for implementation during 
the Full Scale Development and Production phases. 


xvi 


I. INTRODUCTION 


A. BACKGROUND 

The Republic of Korea Air Force (ROKAF) Center has been studying 
military-force increments within the context of constraints imposed by shortfalls in 
the ROKAF budget allocations. The larger context is that the Korean 
governmental budgetary deficit acts as a severe constraint on the general ability 
to fund both ROK defense and improvements to ROKAF. Specifically, both new 
and ongoing weapons-acquisition programs are under financial pressure. In 
response, ROKAF has encouraged efficient cost evaluation, aiming to employ a 


limited budget wisely. 


Some weapons-acquisition programs already in progress have failed to 
arrive at stated goals, compromising related programs in turn. The chronic 
problem of delays and overruns has boxed the ROKAF into attempting the 
development and fielding of ever-more-advanced capabilities, subject to 
increasingly rigid cost and time constraints. Despite these budgetary pressures, 
the master plan for defense-weapons acquisition can be expected to meet 
challenges such as cost increases and scheduling delays if cost evaluation is 
included at the beginning of each component program and as an ongoing part of 
the analysis of each program. Before approval is given for final investment in 
the Korean master weapon-acquisition program, a detailed cost estimation 


should be in place. 


As a case in point, the ROKAF has invested in the development of the T- 
50 training aircraft from concept through development, beginning at the end of 
1992. The T-50, a next-generation training aircraft built by Korea Aerospace 
Industries (KAI) and Lockheed Martin Aeronautics Company, features the power, 
performance, avionics, and cockpit display of the most sophisticated defense 
systems. Using the production of this aircraft to illustrate the value of cost and 


schedule estimation will demonstrate to ROKAF headquarters the importance of 
evaluating a weapon-acquisition program and offer insights towards application 
to other programs. 
B. OBJECTIVE 

The objective of this thesis is to investigate the feasibility of applying a 
knowledge-based approach to the evaluation of a major defense-acquisition 
program- that of the ROKAF T-50 aircraft and to investigate options for adapting 
the US GAO methodology to the ROKAF T-50 aircraft program. Specifically, this 
thesis demonstrates how a knowledge-based approach to the problems of the 
master defense-weapon acquisition program can be defined and employed, and 
offers practical risk-assessment tools to weapons-acquisition officers. 
C. RESEARCH QUESTIONS 

This thesis will investigate the following questions: 


e Can U.S. GAO methodology be applied to the ROKAF T-50 
program? 


e What are the differences between the U.S. and ROKAF programs, 
as measured by GAO methodology? 


° What can we learn about the forecast of success and cost overruns 
in ROKAF programs? 


D. SCOPE AND LITERATURE REVIEW 
1. Scope 
This thesis examines the prospects of success and the risk of cost 


overruns in a ROKAF T-50 aircraft develooment-and-procurement program. 


There are two possible, and different, approaches to examining forecasted 
costs associated with a weapons-acquisition program: 


e Focus on the affordability of system development, using the work of 
the advanced cost-estimating integrated-product team (IPT), as 
was done in the Joint Strike-Fighter (JSF) program. The IPT is 
then tasked with and responsible for developing methods to assess 
program initiatives. 


e Use GAO methodology to assess technology maturity based on 
three knowledge points. 


Because the GAO’s knowledge-based methodology provides the greater 


insight into risks of cost overrun, this thesis applies the GAO approach. 
2 


Discovery of whether a knowledge-based approach can be applied directly to a 
ROKAF acquisition case will ultimately give officers in charge another tool to 
enhance their ability to plan appropriately for projects and procurement. 
Developing and advancing the US programs will be used in comparison with the 
ROKAF T-50 program. 


This thesis requires the use of T-50 program data and assessment of 
technical maturity on the basis of NASA technology-readiness levels (TRLs) _ It 
also implements the latest acquisition-report information available. As part of a 
data-quality assurance-and-validation program, we sought data whose reliability 


was based upon 


e Eliminating date that was anomalous or incredible 
° Reliable sources, such as program managers 
2. Literature Review 


The US GAO has written extensively on ways to enhance the weapon 


acquisition process. For example, 


Although the weapons that the Department of Defense (DOD) 
develops have no rival in superiority, there still remain ways in 
which they can be improved. GAO’s reviews over the past 20 years 
have found consistent problems with weapon acquisitions-cost 
increases, schedule delays, and performance shortfalls-along with 
underlying managers to promise more than they can deliver. DOD 
can resolve these problems by using a knowledge-based approach 
derived from the best practices of successful product 
developments. 


As essential parts of successful program management, the best weapons- 
acquisition programs take extra steps to confirm technology maturity, design 
stability, and production stability. 

Separating technology development from product development is 


important to this effort. Successful programs make a science and 
technology organization, rather than the program or product 


1 GAO-04-248, Assessments of Major Weapon Programs, p. 2. 
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development manager, responsible for maturing technologies. Such 

steps can help to reduce costs and deliver a product on time and 

within budget.?: 

3. Organization and Chapter Summary 

This section gave an overview of the thesis and the process of assessing 


technical maturity based on a GAO report. 


e Chapter Il presents information necessary to understanding 
theoretical cost analysis, the GAO methodology, and statistics. 

e Chapter Ill looks at application of theory and compares data 
pertaining to the U.S. and T-50 master acquisition programs. 

e Chapter IV provides conclusions and recommendations for further 
research. 


2 GAO 05-301, Assessments of Selected Major Weapon Programs, p. 6. 
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ll. RELATED RESEARCH 


A. WEAPON ACQUISITION ENVIRONMENT 

1. Introduction 

The program manager also should consider the requirements of users in 
various combatant services. Though weapon acquisition program managers and 
decision makers have been struggling to achieve desirable technical levels within 
schedule and budget, the acquisition process has proved somewhat 
unpredictable. For example, 


the US weapons-acquisition program results in reduced quantities 
and increased costs. The JSF acquisition program’s estimated 
development and procurement costs have increased. In addition, 
the number of aircraft it plans to deliver has been reduced. As a 
result, unit costs for the JSF aircraft have increased substantially, 
thereby reducing the program’s buying power. The most significant 
quantity reduction occurred after system development began in 
2001, when the program reduced the number of aircraft it plans to 
procure from 2,852 to 2,443, or by 14 percent. The Navy-concerned 
that it could not afford the number of tactical aircraft it planned to 
purchase-reduced the number of JSF aircraft for joint Navy and 
Marine Corps operations from 1,089 to 680 by reducing the number 
of backup aircraft needed.? 


2. The T-50 Program 


The T-50 is a true digital aircraft with the power, performance, 
avionics and cockpit display of today's most sophisticated defense 
systems. Everything about the T-50 readies student pilots for the 
rigors of frontline fighters. Through the joint development team of 
Korea Aerospace Industries and Lockheed Martin Aeronautics 
Company, an all-new trainer bridges the gap between subsonic 
training and supersonic, next-generation fighters. The T-50 easily 
exceeds Mach 1 and offers the widest flight envelope of any 
advanced jet trainer (AJT) and lead-in fighter trainer (LIFT) 
available. Pilots take the controls of a smooth-handling jet that 
offers the exceptional maneuverability of next-generation fighters, 
with similarly integrated cockpit and systems.4 


3 GAO 05-271, Joint Strike Fighter Acquisition, p. 7. 
4 ROK Air Force, http://www.airforce.mil.kr/ENG/index.html (Accessed October 24, 2005) 
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3. The US Acquisition Programs 
This thesis will use the US programs for comparison with the T-50 
program. A brief description to understand the general of each program follows. 


. B-2 Radar Modernization Program (RMP): “The Air Force’s B-2 
RMP is designed to modify the current radar system to resolve 
potential conflicts in frequency band usage. To comply with federal 
requirements, the frequency must be changed to a band where the 
B-2 will be designated as a primary user. The modified radar 
system is being designed to support the B-2 stealth bomber and its 
combination of stealth, range, payload, and near precision weapons 
delivery capabilities.”© 


e C-130 Avionics Modernization Program (AMP): “The Air Force’s C- 
130 AMP standardizes the cockpit configurations and avionics for 
14 different mission designs of the C-130 fleet. It consolidates and 
installs the mandated DOD Navigation/Safety modifications, the 
Global Air Traffic Management systems, and the C-130 broad area 
review requirements. It also incorporates other reliability, 
maintainability, and sustainability upgrades and provides increased 
situational awareness capabilities and reduces susceptibility of 
Special Operations aircraft to detection/interception.”® 


e C-5 Avionics Modernization Program (AMP): “The Air Force’s C-5 
AMP is the first of two major upgrades for the C-5 to improve the 
mission capability rate, transport capabilities and reduce ownership 
costs. The AMP implements Global Air Traffic Management, 
navigation and safety equipment, modern digital equipment, and an 
all-weather flight control system.”7 


e C-5 Reliability Enhancement and Reengineering Program (RERP): 
“The Air Force’s C-5 RERP is one of two major upgrades for the C- 
5. The RERP is designed to enhance the reliability of the aircraft 
through the replacement of engines and modifications to 
subsystems such as the electrical, fuel, hydraulic and flight controls 
systems, while the C-5 Avionics Modernization Program (AMP) is 
designed to enhance the avionics. These upgrades are part of a 
two-phased modernization effort to improve the mission capability 
rate, transport capabilities and reduce ownership costs.”8 


e F/A-22 Raptor: “The Air Force’s F/A-22, originally planned to be an 
air superiority fighter, will also have air-to-ground attack capability. 
It is being designed with advanced features, such as stealth 





5 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 31. 
6 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 33. 
7 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 35. 
8 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 37. 
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characteristics, to make it less detectable to adversaries and 
capable of high speeds for long ranges. It also has integrated 
aviation electronics (avionics) designed to greatly improve pilots’ 
awareness of the situation surrounding them. It is designed to 
replace the Air Force’s F-15 aircraft.”9 


Global Hawk: “The Air Force’s Global Hawk system is a high 
altitude, long endurance unmanned aerial vehicle with integrated 
sensors and ground stations providing intelligence, surveillance, 
and reconnaissance capabilities.”1° 


Joint Strike Fighter (JSF): “The program goals are to develop and 
field a family of stealthy, strike fighter aircraft for the Navy, Air 
Force, Marine Corps, and U.S. allies, with maximum commonality 
to minimize costs. The carrier suitable version will complement the 
Navy’s F/A-18 E/F. The conventional take-off and landing version 
will primarily be an air-to-ground replacement for the Air Force’s F- 
16 and A-10 aircraft, and will complement the F/A-22. The short 
take-off and vertical landing version will replace the Marine Corps’ 
F/A-18 and AV-8B aircraft.”"1 


Joint Unmanned Combat Air Systems (JUCAS): “The program is a 
combined effort of the Defense Advanced Research Projects 
Agency (DARPA), the Air Force, and the Navy to demonstrate the 
technical feasibility and operational value of a networked system of 
high performance and weaponized unmanned air vehicles. 
Expected missions include the suppression of enemy air defenses, 
electronic attack, precision strike, and surveillance. The program 
consolidates two formerly separate service projects and is to 
develop larger, more capable, and interoperable aircraft.”12 


Predator: “The Air Force’s MQ-9 Predator B is a multi-role, 
medium-to-high altitude endurance unmanned aerial vehicle 
system capable of flying at higher speeds and higher altitudes than 
its predecessor the MQ-1 Predator A. The Predator B is designed 
to provide a ground attack capability and will employ fused multi- 
spectral sensors to find and track small ground mobile or fixed 
targets. As envisioned, each Predator B system will consist of four 
aircraft, a ground control station, and a satellite communication 
suite.”18 





9 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 63. 


10 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 67. 


11 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 79. 


12 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 87. 


13 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 101. 
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e V-22 Joint Services Advanced Vertical Lift Aircraft: “The V-22 
Osprey is a tilt rotor, vertical takeoff and landing aircraft being 
developed by the Navy for Joint Service application. It is designed 
to meet the amphibious/vertical assault needs of the Marine Corps, 
the strike rescue needs of the Navy, and the special operations 
needs of the Air Force and the U.S. Special Operations Command. 
The MV-22 version will replace the CH-46E and CH-53D 
helicopters of the Marine Corps.”14 


4. The Technology Development Phase 

The figure below’S show the phases of the weapon system development 
and acquisition process, as described in DoDI 500.2. The Technology 
Development Phase is a specific phase of this process, and the following 
descriptions,'® in Table 1, provide a more detailed description of the Technology 
Development Phase. 





User Neeps & 
TECHNOLOGY OpPorRTUNITIES 





ft ™ 







TeEcHNOLoGy | System DEVELOPMENT PRODUCTION & 
& DEMONSTRATION DEPLOYMENT 


Design 
Readiness 
Review 


Concept 


Concept 
Decision 







Figure 1. The Technology Development Phase(From: DoD 5000.2, Defense 
Acquisition Guidebook, - 3. Procedures) 





14 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 117. 
15 DoD 5000.2, Defense Acquisition Guidebook, - 3. Procedures, Figure 1, Framework. 
16 DoD 5000.2, Defense Acquisition Guidebook, - 3.6. Technology Development. 


8 





a. Purpose 





The purpose of the technology-development phase is to reduce technology risk and to determine 
the appropriate set of technologies to be integrated into a full system. Technology Development 
is a continuous technology discovery and development process reflecting close collaboration 
between the S&T community, the user, and the system developer. It is an iterative process 
designed to assess the viability of technologies while simultaneously refining user requirements. 





b. Milestone A 





The project shall enter technology development at Milestone A when the MDA has approved the 
Technology Development Strategy (TDS). The tables in Enclosure 3 identify all statutory and 
regulatory requirements applicable to Milestone A. This effort normally shall be funded only for 
the advanced development work. A favorable Milestone A decision DOES NOT mean that a new 
acquisition program has been initiated. 





Cc. Shipbuilding programs 





Shipbuilding programs may be initiated at the beginning of Technology Development. The 
information required in the tables at Enclosure 3 shall support program initiation. A cost 
assessment shall be prepared in lieu of an independent cost estimate (ICE), and a preliminary 
assessment of the maturity of key technologies shall be provided. 





d. Technology consideration 





The Interim Capability Document (ICD) and the Technology Development Strategy (TDS) shall 
guide this effort. Multiple technology-development demonstrations may be necessary before the 
user and developer agree that a proposed technology solution is affordable, militarily useful, and 
based on mature technology. The TDS shall be reviewed and updated upon completion of each 
technology spiral and development increment. Updates shall be approved to support follow-on 
increments. 





e. Evolutionary acquisition 





If an evolutionary strategy is used, the initial capability represents only partial fulfillment of the 
overall capability described in the Interim Capability Document (ICD), and successive technology 
development efforts continue until all capabilities have been satisfied. In an evolutionary 
acquisition, the identification and development of the technologies necessary for follow-on 
increments continues in parallel with the acquisition of preceding increments, allowing the mature 
technologies to more rapidly proceed into System Development and Demonstration (SDD). Each 
increment of an evolutionary acquisition program shall have an associated MDA-approved TDS. 





f. Exiting the phase 





The project shall exit technology development when an affordable increment of militarily useful 
capability has been identified, the technology for that increment has been demonstrated in a 
relevant environment, and a system can be developed for production within a short timeframe 
(normally less than five years); or when the MDA decides to terminate the effort. During 
Technology Development, the user shall prepare the Capability Development Document (CDD) to 
support program initiation, refine the integrated architecture, and clarify how the program will lead 
to joint war fighting capability. The CDD builds on the ICD and provides the detailed operational 
performance parameters necessary to design the proposed system. A Milestone B decision 
follows the completion of Technology Development. 











Table 1. Detailed Description of the Technology Development Phase (From: 
DoD 5000.2, Defense Acquisition Guidebook, - 3.6. Technology 
Development) 
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B. COST ESTIMATING"” 


Cost analysis is: (1) the act of developing, analyzing, and 
documenting cost estimates using analytical approaches and 
techniques. (2) The process of analyzing and_ estimating 
incremental and total resources required to support past, present, 
and future forces, units, systems, functions, and equipment. _ It is 
an integral step in the selection between alternatives by the 
decision maker. (3) A management tool used to help decision 
makers evaluate resource requirements at key management 
milestones and decision points in the acquisition process. Cost 
analysis is used to produce cost estimates for materiel systems, 
automated information systems, force units, training, and other 
Army programs and projects. Each cost analysis should contain: 


(1) aclear definition of what is being costed. 


(2) The specification of all assumptions, ground rules, and 
constraints, assumed or imposed, underlying the analysis. 
They must each be explained with adequate rationale. 


(3) An estimate of all expected costs, directly or indirectly 
associated with the project over its life, including disposal. 
The cost estimate must include the identification of all data 
sources used. 


(4) — Risk and uncertainty analyses identifying any circumstances 
which could affect a course of action. 


(5) Key limitations in terms of elements that were excluded. 


The documentation supporting the cost analysis should describe 
the methodology used in developing these estimates. It also 
should identify all the data sources and include the computations 
used to estimate the costs. The documentation should be in 
sufficient detail to permit reviewers to follow the logic from 
assumptions to conclusion and to update the estimate at a later 
time. 


Cost analysis is a critical element in the DoD acquisition process. 
It supports management decisions by quantifying the resource 
impact of alternative options. A quality analysis includes different 
acquisition strategies, hardware designs, software designs, 
personnel requirements, and operating and support concepts. As 
a program matures and more information becomes available, the 
cost estimate grows in complexity and detail. One test of the utility 


17 Department of the Army Cost Analysis Manual, U.S. Army Cost and Economic Analysis 
Center, May 2002. 
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of cost analysis is its ability to respond quickly to program 
turbulence. Army planners must have reliable and _ readily 
available information about the cost consequences of program 
changes, extensions, or cancellations. | Cost analysts must 
develop models to support these quick turnaround analyses. 


Cost analysis has an on-going role in the management of base 
operations as well as acquisitions. Cost analysis assists 
installations, headquarters in determining base — support 
requirements, developing budgets, conducting cost benefit 
analysis, and performing special studies. For example, the office of 
the Deputy Assistant Security of Army Cost and Economics (DASA- 
CE) develops cost factors in support of the Army Chief of Staff for 
Installation Management (ACSIM) for both the Installation Status 
Report (ISR) and the Army Installation Management - Headquarters 
Information (AIM-HI) model. Other ACSIM efforts supported by 
cost analysis include A-76 studies, Service Based Costing, and 
Standard Service Costing. 


With the establishment of the cost/outcome oriented Government 
Performance Results Act (GPRA), cost analysis has taken on a 
larger role in to support management of base operations. The 
managerial costing focus, to meet GPRA mandates, requires cost 
analysis in the measuring and management of cost and results. 
Cost analysis will be needed to develop methodologies, conduct 
studies and analyze data of the products and services provided 
through base operations. The prerequisite to cost management is 
cost measurement. There are numerous methods of measuring 
costs, all of which will require cost analysis skills now and in the 
future. Examples of cost measurement include, full cost, job-order 
cost, service based cost, activity based cost, standard cost, product 
cost, and responsibility cost to name a few. Though there are 
many examples of cost measurement each demands cost analysis 
support to make information meaningful to DoD management. 
DASA-CE will prepare a managerial costing manual in the future on 
Activity Based Costing, Service Based Costing and Standard 
Service Costing. 


Other uses of cost analysis in the DoD are to: 


(1) |Support decisions on program viability, structure, and 
resource requirements. 


(2) | Evaluate the cost implications of alternative materiel system 
designs. 


| 


(3) | Provide credible and auditable cost estimates in support of 
milestone reviews during the acquisition process. 


(4) Assess the cost implications of new technology, new 
equipment, new force structures, or new operating or 
maintenance concepts. 


(5) Support the Planning, Programming, Budgeting, and 
Execution System (PPBES) process. This includes 
formulating and documenting Army Cost Positions (ACPs) 
on programs within the Program Objective Memorandum 
(POM) and the Budget Estimate Submission (BES) 
processes. 


(6) | Determine the funds required for a given level of training or 
operational activity such as miles driven per year. 


Cost analysis applies scientific and statistical methods to evaluate 
the likely cost of a specific item in a defined scenario. In the real 
world, there are multiple uncertainties about the item's cost. Some 
"internal" uncertainties influencing cost are inadequate item 
definition, poor contract statement of work, optimistic proposed 
solutions, inexperienced management, and_ success-oriented 
scheduling. Some "external" uncertainties include funding 
turbulence, contractor's underestimating of complexity, contractor's 
changing business base, and _ excessive (or _ insufficient) 
Government oversight. In spite of uncertainty, the process of cost 
analysis is the most rigorous approach available to evaluate the 
costs of alternatives for the decision maker. Cost analysis does 
have limitations. Analysts develop cost estimating methodologies 
with an imperfect understanding of the technical merits and 
limitations of the item. The applicability of historic data is always 
subject to interpretation. Because of future uncertainties, there 
are limitations in determining the degree to which reality varies from 
the plan. Realistically, the cost analysis process cannot: 


(1) | Be applied with cookbook precision, but must be tailored to 
the problem. 


(2) Produce results that are better than input data. 
(3) Predict political impacts. 
(4) Substitute for sound judgment, management, or control. 


(5) Make the final decisions. 
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Despite these limitations, cost analysis is a powerful tool. 
Rigorous and systematic analysis leads to a better understanding 
of the problem. It improves management insight into resources 
allocation problems. Because the future is uncertain our best 
estimate will differ from reality. 


C. GAO METHODOLOGY 

Our review of the “best practices” literature in weapons-acquisition 
programs provides the lesson that three considerations are critical for sound 
progress in the weapons-acquisition process. These are 

e Technology maturity 

e Design stability 

e Controlled production stability 

Technology development is the key factor in determining whether a 
weapons-development program could meet requirements without cost overruns 
and schedule. Acquisition-program management also relies on the knowledge- 
based approach in developing new weapon programs and ensures key results 


through high levels of knowledge. 


The US GAO methodology implements three knowledge points, as 
discussed below, and the GAO writes that 
The attainment of each successive knowledge point builds on the 
preceding one. While the knowledge itself builds continuously 
without clear lines of demarcation, the attainment of knowledge 
points is sequential. In other words, production maturity cannot be 
attained if the design is not stable, and design stability cannot be 
attained if the critical technologies are not mature.18 
These knowledge points are described in the paragraphs below. 
1. Knowledge Point One: Technology 
Technology Readiness Levels (TRLS), used in the US_ National 
Aeronautics and Space Administration’s (NASA) planning for many years, are 
measured on a scale of nine levels of technological maturity, as shown in Table 
2.19 Knowledge point one occurs when the weapon-acquisition requirements 


18 GAO 05-301, Assessments of Selected Major Weapon Programs, p. 7. 


19 Technology Readiness Level Definitions, 
http://www.globalsecurity.org/military/intro/trl.htm (accessed November 29, 2005) 
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meet the developer's available resources in terms of knowledge, time, and 
budget. The availability of mature technology on which to base development 
provides enhanced probability of achieving project success from the beginning. 
TRLs have been adapted to facilitate comparison between TRLs at the 
acquisition level and TRLs at the parts level. To identify critical levels of 
technology, GAO attempts to find the key technologies in which information 
availability may be a concern. As shown in the GAO report, “our best practices 
work has shown that a technology readiness level of 7—demonstration of a 
technology in an operational environment—is the level of technology maturity 


that constitutes a low risk of starting a product development program.”2° 





Technology 
Readiness Level 


Description 














1. Basic principles Lowest level of technology readiness. Scientific research begins with to be 
observed and translated into applied research and development. Example might include paper 
reported studies of a technology's basic properties. 

2. Technology concept Invention begins. Once basic principles are observed, practical applications can 
and/or application be invented. The application is speculative and there is no proof or detailed 
formulated analysis to support the assumption. Examples are still limited to paper studies. 

3. Analytical and Active research and development is initiated. This includes analytical studies 
experimental critical and laboratory studies to physically validate analytical predictions of separate 
function and/or elements of the technology. Examples include components that are not yet 
characteristic integrated or representative. 

4. Component and/or Basic technological components are integrated to establish that the pieces will 
breadboard work together. This is relatively "low fidelity" compared to the eventual system. 
validation in Examples include integration of 'ad hoc' hardware in a laboratory. 
laboratory 


environment 








5. Component and/or Fidelity of breadboard technology increases significantly. The basic 
breadboard technological components are integrated with reasonably realistic supporting 
validation in relevant elements so that the technology can be tested in a simulated environment. 
environment Examples include ‘high fidelity’ laboratory integration of components. 

6. System/subsystem Representative model or prototype system, which is well beyond the 
model or prototype breadboard tested for TRL 5, is tested in a relevant environment. Represents a 
demonstration in a major step up in a technology's demonstrated readiness. Examples include 
relevant testing a prototype in a high fidelity laboratory environment or in simulated 
environment operational environment. 


20 GAO 05-301, Assessments of Selected Major Weapon Programs, p. 130. 
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Technology i 











7. System prototype Prototype near or at planned operational system. Represents a major step up 
demonstration in a from TRL 6, requiring the demonstration of an actual system prototype in an 
operational operational environment, such as in an aircraft, vehicle or space. Examples 
environment include testing the prototype in a test bed aircraft. 

8. Actual system Technology has been proven to work in its final form and under expected 
completed and ‘flight conditions. In almost all cases, this TRL represents the end of true system 
qualified’ through development. Examples include developmental test and evaluation of the 
test and system in its intended weapon system to determine if it meets design 
demonstration specifications. 

9. Actual system ‘flight Actual application of the technology in its final form and under mission 
proven’ through conditions, such as those encountered in operational test and evaluation. In 
successful mission almost all cases, this is the end of the last "bug fixing" aspects of true system 
operations development. Examples include using the system under operational mission 

conditions. 

Table 2. Definition of Technology Readiness Level (From: 
http://www.globalsecurity.org/military/intro/trl.htm, accessed November 29, 
2005) 

2. Knowledge Point Two: Design 


This knowledge point occurs at the point when a product’s design 
drawings are released from the engineer to the manufacturer. Successful 
programs obtain design stability at the time of critical design review (CDR), and 
CDR results serve as criteria for estimating the stability of engineering drawings 
released. The benchmark that the US GAO has set is that engineering-drawing 
stability is achieved if it is true that at least 90% of engineering drawings are 
completed (or at least releasable) before the CDR. 

3. Knowledge Point Three: Production 

Products manufactured with statistical process controls in place to govern 
costs, and schedules represent maturity in the production process. To evaluate 
such maturity, critical manufacturing processes are monitored for statistical data 
and fractions of statistical process control. To ascertain status of production, two 
concepts should be considered: quality and product testing. 


° Quality in the manufacturing processes requires a repeatable 
statistical approach to ensure that standards for the system have 
been defined and are being monitored. Production must be on time, 
meet schedules, and keep in cost. 
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e Product test is needed to guarantee that the system operates 
reliably, without failure or repair during the some period. Production 
tests require many trials and processes. 


Due to the typical shortage and high cost of the personnel needed to 
perform such tests, we focus only on the critical processes necessary to meet the 
reliability and performance standards. To measure the performance of the 
process, a process-capability index (CPK) is used to determine how a process is 
running compared to its specification limits and to measure production-process 
capability within specified limits. 

GAO used the Process Capability Index which is a process 

performance measurement that quantifies how closely a process is 

running to its specification limits. The index can be translated into 

an expected product defect rate, and GAO has found it to be a best 

practice.2! 

The CPK and probability of a defective part are shown in the table 
below.22 A standard of less than 1.33 indicates that the manufacturing process 
lacks statistical control and acceptable consistency. Satisfactory cost, schedule, 
and product quality could be obtained during the manufacturing process prior to 


























production. 

Manufacturing process capability index | Associated defect rate 

Cpk - .67 (not capable) 1 in 22 parts produced 

Cpk — 1.0 (marginally capable) 1 in 370 parts produced 

Cpk - 1.33 (industry standard) 1 in 15,152 parts produced 

Cpk — 2.0 (industry growth goal) 1 in 500,000,000 parts produced 
Table 3. CPK and Probability of a Defective Part (From: GAO 02-701) 


21 GAO 05-301, Assessments of Selected Major Weapon Programs, p. 131. 
22 GAO 02-701, Best Practices, p. 39. 
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a. Process Capability Index23 

Process capability compares the output of an in-control process to 
the specification limits by using capability indices. The comparison is made by 
forming the ratio of the spread between the process specifications (the 
specification "width") to the spread of the process values, as measured by six 
process standard-deviation units (the process "width"). A process capability index 
uses both the process variability and the process specifications to determine 
whether the process is "capable." A capable process is one where almost all the 
measurements fall inside the specification limits. This can be represented 


pictorially by the plot below. 





LSL ry 4 USL 





actual process spread 


SEW able procass Spread 


Figure 2. The Specification Limits (From: 


http://www. itl.nist.gov/div898/handbook/pmc/section1/pmc16.htm 
accessed November 29, 2005) 


There are several statistics that can be used to measure the 
capability of a process: Cp, Cox, Com. 


Most capability indices estimates are valid only if the sample size 
used is large enough, generally thought to be about 50 independent data values. 


23 Engineering Statistics Book, 
http://www. itl nist.gov/div898/handbook/pmc/section1/pmc16.htm (accessed November 29, 2005). 
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The Cp, Cpx, and Com statistics assume that the population of data 
values is normally distributed. Assuming a two-sided specification, if ys and a 
are the mean and standard deviation, respectively, of the normal data and USL, 
LSL, and T are the upper and lower specification limits and the target value, 
respectively, then the population capability indices are defined as follows: 





USL —LSL 
Cy = ba 
__, USi—p p- Loi 
Ga = min| 30” Ban 
USE, — ELSE 


a 6,/o7 + (a —T)? 


Sample estimators for these indices are given below. (Estimators 
are indicated with a "hat" over them). 


, _USL—LSL 


- 6s 
A _ Ueb—F E—-LSL 
Cyx = min > —= 
, ._ USL—LSL 


6/9? + (% —TY? 


The estimator for C>. can also be expressed as C,x = C,(1-k), 


where k is a scaled distance between the midpoint of the specification range, m, 
and the process mean, #4. 


Denote the midpoint of the specification range by m = 


(USL+LSL)/2. The distance between the process mean, /4, and the optimum, 
which is m, is pm, where m “ ss < LS. The scaled distance is 
[re — | 


ee ee 
(USL — LSL)/2” 


Q<k<1 
(the absolute sign takes care of the case when LSL < ji = mt). To determine 
the estimated value, & we estimate ys by #. Note that # < USE. 
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The estimator for the Cp index, adjusted by the k factor, is 


pe = G,(1 — f) 


Since 0 < & <1, it follows that Cy, < Cy, 


To get an idea of the value of the Cp statistic for varying process 


widths, consider the following plot: 
























































USL - LSL 6a 8a 10a 120 
Cp 1.00 1.33 1.66 2.00 
Rejects .27% 64 ppm .6 ppm 2 ppb 
% of spec used 100 75 60 50 











Figure 3. 
http://www. itl _nist.gov/div898/handbook/pmc/section1/omc16.htm 


te, 





Varying Process Widths (From: 


accessed November 29, 2005)) 











This can be expressed numerically by the table below where ppm = 
parts per million and ppb = parts per billion. Note that the reject figures are based 
on the assumption that the distribution is centered at #. 


We have discussed the situation with two spec limits: the USL and 
LSL. This is known as the bilateral or two-sided case. There are many cases 
where only the lower or upper specifications are used. Using one spec limit is 
called unilateral or one-sided. The corresponding capability indices are 


allowable upper spreacl = USL — js 


G “= actual upper spread 3a 


and 


C,= allowable lower spread  ys— DSL 
“actual lower spread 30 


where sa and @ are the process mean and standard deviation, respectively. We 
would like to have Cok at least 1.0. Below 1.0 means this is not a good 
process. If possible, reduce the variability or/and center the process. We can 


compute the c and Ci. Estimators of Cpu and Cy; are obtained by replacing 


fs and ao by 2 and s, respectively. The following relationship holds Cy = (Cpu + 
Ci) /2. 


This can be represented pictorially by: 
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: A WSL 


UA! Upper Spread 


allowable yppey spread 


Figure 4. Actual and Allowable Upper Spread (From: 


http://www. itl nist.gov/div898/handbook/pmc/section1/pmc16.htm 
accessed November 29, 2005)) 


Note that we also can write: Cpx = min {Cp, Coup.” 


b. Manufacturing and Product Reliability Knowledge of 
GAO Report 


The decision maker must decide whether the weapons-acquisition 
program is ready to make the transition from development into production. The 
below Table 424 would provide the guidance necessary to making a successful, 
data-driven, decision process in the weapon-acquisition program, thereby 
reducing the possibility of delaying the schedule and increasing costs. 





Identify key system Key product characteristics and critical manufacturing 
characteristics and processes are identified. Because there can be thousands of 
critical manufacturing manufacturing processes required to build a product, 
processes. companies focus on the critical processes—those that build 


parts that influence the product’s key characteristics such as 
performance, service life, or manufacturability. 





Determine processes in Statistical process control is used to determine if the 

control and capability. processes are consistently producing parts. Once control is 
established, an assessment is made to measure the process’s 
ability to build a part within specification limits as well as how 
close the part is to that specification. A process is considered 
capable when it has a defect rate of less than one out of every 














24 GAO 02-701, Best Practices, p. 38. 
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15,152 parts produced. 





Conduct failure modes Bottom-up analysis is done to identify potential failures for 
and effects analysis. product reliability. It begins at the lowest level of the product 
design and continues to each higher tier of the product until 
the entire product has been analyzed. It allows early design 
changes to correct potential problems before fabricating 





hardware. 
Set reliability growth A product’s reliability is its ability to perform over an expected 
plan and goals. period of time without failure, degradation, or need of repair. A 


growth plan is developed to mature the product’s reliability 
over time through reliability growth testing so that it has been 
demonstrated by the time production begins. 





Conduct reliability Reliability growth is the result of an iterative design, build, test, 
growth testing. analyze, and fix process for a product’s design with the aim of 
improving the product’s reliability over time. Design flaws are 
uncovered and the design of the product is matured. 





Conduct executive level | Corporate stakeholders meet and review relevant product 
review to begin knowledge, including manufacturing and reliability knowledge, 
production. to determine whether a product is ready to begin production. 
The decision is tied to the capture of knowledge. 














Table 4. Activities to Capture Manufacturing Knowledge and Make 
Decisions (After: GAO 02-701) 


D. EVALUATING AFFORDABILITY INITIATIVES 

There are different approaches to determining technology maturity. For 
example, there are those developed by the Joint Strike Fighter (JSF) Program 
Office’s (PO) Advanced Cost Estimating (ACE) Integrated Product Team (IPT) to 
clarify their technology initiatives. The description below25 shows the status and 
definition of technology maturity factors, as well as the sub-factors26 that support 


them. 


Definitions for maturity factors were considered carefully by IPT 
members. All scales range from very easy (a known, presently 
operational technology, design, or process) to extremely difficult 
(purely theoretical concept or laboratory research yet to be 
attempted in a production environment). 


The state-of-technology sub-factor measures each initiative in 
terms of availability and promise of the technology required for 


25 Johns Hopkins APL Technical Digest, volume21, number 3 (2000), Evaluating 
Affordability Initiatives, pp. 429-430. 


26 Johns Hopkins APL Technical Digest, volume 21, number 3 (2000), Evaluating 
Affordability Initiatives, p. 430. 
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success. Ability to attain the required level of technological 
sophistication within known schedule constraints, as well as 
hardware and _ testing maturity, is considered. Although 
assessments were performed on a “system” basis, participants 
were also asked to consider and comment on the technological 
requirements of key subsystems. The scale ranges from technology 
that is already operational and deployed to that which still requires 
significant scientific research 


The design and engineering sub-factor measures each initiative in 
terms of difficulty in advancing the state of the art to that required 
for the JSF Program. Thus, design and engineering focuses on 
implementation, separate from technology. For example, an item 
previously engineered and_ production-qualified may require 
extensive form, fit, and function changes or modifications for JSF 
application. Key subsystems are also considered. The scale ranges 
from off-the-shelf items meeting all requirements to ones requiring 
new, breakthrough design or engineering efforts. 


The manufacturing process sub-factor measures each initiative in 
terms of the process capability needed to produce required 
quantities for the JSF Program. Process attributes considered are 
metrics such as number of allowable defects, yield or throughput 
requirements, tolerance or precision requirements, overall process 
capability to be maintained, and allowable failure rates. Evaluation 
includes assessment of advances required to move from current to 
proposed manufacturing and assembly processes. The scale 
ranges from an existing demonstrated process that satisfies all key 
attributes to one that exceeds the state of the art for at least one 
key attribute. 


The resource availability sub-factor grades an initiative on the 
availability of all resource elements necessary for implementation at 
production quantities and rates specified for the JSF Program. 
Resources include parts or subassemblies, tools and fixtures, test 
equipment and facilities, personnel (including their skills and 
training levels), materials (quantity and quality), production 
equipment and facilities, and funding. The scale ranges from all 
required resources being readily available through at least one 
dependable source to an initiative for which resources have not yet 
been specified. 
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State of Design and Manufacturing Pena ree 
Level : : Resource availability 
technology engineering process 
initiative “Off-the shelf Existing, Readily obtainable through at 
presently : 
A 2 hardware demonstrated least one source; successful 
operational and ; : 
requirements process past experience 
deployed 
Design Bahia 
Process in required; Modified; All but one resource 
B limited existing witli obtainable from at least one 
; demonstrated 
operation components, source 
norms 
specs 
Process passed Design New 
acceptance; haere ee All but two resources 
required; combination of : 
C approved for obtainable from at least one 
aus beyond demonstrated 
limited source 
present specs | processes 
operation 
Prondsspaseed Some Demonstrated, More than two resources 
D ealneciien fasts development but one key obtainable from at lest one 
" effort required attribute new source 
Process passed 
performance Moderate Demonstrated, Onions tescurce obtainable 
E tests/in development but two or more y 
ea é from at least one source 
qualification effort required new attributes 
testing 
Process Major New process; 
ei ae Resources have been 
F feasibility development within eseciied 
demonstrated effort required state of the art ss 
Prototype system 
tested; significant Breakthrough” | New process; Resourses Have Hol been 
G scientific development exceeds state of 





research 
required 





effort required 





the art 





specified 





* Resource include tools/futures, test equipment, personnel, materials, and facilities. 


Table 5. 


Definitions of Maturity Sub Factors (From: Johns Hopkins APL 


Technical Digest, volume 21, number 3 (2000), Evaluating Affordability 


Initiatives) 
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E. SUMMARY 

This chapter highlighted the features of the related research. The 
technology-development phase reflecting the technology discovery processes, 
addresses technology risk and stage of development needed to satisfy the user 
requirements. GAO methodology also suggests the three points in time at which 
to judge technology maturity, and these judgments play a significant role in 
judging the technological risk in the weapons-acquisition program. 


° First, estimate the technology maturity from the development start 
to design review and production. The technology readiness levels 
originally developed by the National Aeronautics and Space 
Administration may be used as criteria. 


e The second technology part, design stability, will be estimated by 
the releasable design drawings at the critical design review. 
Releasable design drawings mean the completed engineering 
drawings, which can be released to the manufacturing stage. 


e The third, and last, technology part, production maturity, means that 
critical manufacturing processes are under control of statistical 
process and the Process Capability Index is used to quantify the 
statistical process of the product. 


The manufacturing and product reliability Knowledge of the GAO report 
shows the ideal decision process to meet the schedule, budget limitation, and 
user’s requirements concerning all weapon acquisition program. Yet there are 
different approaches to determining the level of technology maturity by examining 
the affordability of weapons-system development. There is also some criticism 
about GAO methodology. Even though GAO methodology could be used with the 
knowledge-based approach, Department of Defense disagrees with GAO's 
March 05 assessment of JSF technical maturity as stated in GAO reports GAO- 
05-271 and GAO-05-301. Although the knowledge points provide excellent 
indicators of potential risks, they do not by themselves cover all elements of risk 
that a program encounters during development, such as funding instability. Our 
detailed reviews on individual systems normally provide for a fuller treatment of 


risk elements.27 


27 GAO 05-301, Assessments of Selected Major Weapon Programs, p. 132. 
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lil. IMPLEMENTATION AND APPLICATION 


A. TRL ASSESSMENT 

1. Technology Assessment 

Tardy arrival of key technologies can significantly delay the development 
of a weapon system. Testing delays and cost increases in the F-22 program, for 
example, can be partly traced to inadequate maturity of Avionics, as measured 
by TRLs. To prevent a similar outcome in the ROKAF T-50 program, program 
data has been examined and GAO knowledge sets applied, and the resulting 
assessment will be compared against the US program. Because judging TRLs 
may be difficult at best, gathering up-to date information about the program as a 
whole is critical. We requested such data from program managers and made 
use of confidential TRL data provided to NPS about the U.S. program. Institute 
reports on the T-50 will rate the program according to a nine-level technology- 
maturity scale, estimating thirteen critical issues and 649 technological elements 
and applying GAO standards, by which “a technology readiness level of 7- 
demonstration of a technology in an operational environment - is the level of 
technology maturity that constitutes a low risk of starting a product development 
program,”28 to TRL findings. The following TRL of B-2RMP, F-22 and Predator 
were derived by mapping the descriptions of technologies from GAO reports. 
These mappings represent the author’s own views. They are not meant to be an 
endorsement of any official views and others may view the matter differently. The 
others are the reliable data from program manager and reports. The assessment 
of T-50 TRLs comes from “Analytic evaluation on technology acquisition of T-50 


program” and is described in part 2 of this chapter. 


28 GAO 05-301, Assessments of Selected Major Weapon Programs, p. 130. 
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Figure 7. Production TRL 
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2. Technology Uncertainty 

Program managers of weapon acquisition try to judge the level program 
risk by using TRLs. Sometimes, in the rapidly growing technology environment, a 
low level of technology in the beginning of the development phase will be 
sufficient to achieve the desired TRL in the design review and production phases. 
The difficult aspect of using this methodology in this environment is the challenge 
of estimation future TRL. “The higher the level of technological advancement, the 
higher the level of opportunity costs for a successful decision. Greater 
technological complexity implies a greater degree of uncertainty or ambiguity for 
the key decision maker’29 In this respect, the US programs also have to 


accommodate uncertainty in forecasting reasonable, future values of TRLs. 


The data given by US program managers also shows the difficulty of 
estimating the TRL in the design review and production period. The same 
challenge is true in the case of T-50 program. There is some literature which 
provides a method for estimation future values of TRLs. For forecasting the 
maturation of technology through TRLs, we reviewed the report by Kirby and 
Mavris. Unfortunately, this paper applies only to TRL 1~5, which is below the 
range of interest in this thesis. Still, the conclusions of this report are of interest. 


This paper described research in the area of probabilistic 
technology assessments and techniques to forecast the impact of 
any emerging technology in the conceptual and preliminary phases 
of aircraft design. The thrusts of the techniques developed were 
focused on the description of technology development programs, 
and the various milestones encountered during a successful 
program. The identification of sources of uncertainty associated 
with an immature technology were described and applied to the 
determination of frequency distributions of a technology’s impact on 
an aerospace system.?0 


Two broad categories of forecasting exist: exploratory and 
normative. Exploratory forecasting techniques consider historical 
trends and extrapolate into the future to see what may happen. The 





29 A Methodology for Assessing Acquisition Technical Risk, Harrison, William, Gerald, Jr., p. 
62. 


30 Forecasting Technology Uncertainty in Preliminary Aircraft Design, Michelle R. Kirby and 
Dimitri N. Mavris, p. 11. 


30 


feasibility of this process depends upon an assumption that 
progress is evolutionary and does follow a regular pattern. The 
normative method begins with future goals and works backward to 
identify the levels of performance needed to obtain the desired 
goals, if at all achievable with the resources available. Either 
perspective utilizes one, or combinations, of four traditional 
forecasting techniques: S-curves, trend extrapolation, the Delphi 
method, or scenario development. The first two techniques assume 
a functional form of previous technological growth and extrapolate 
to a future time. Again, sufficient information must exist for the 
forecast to be accurate and of value to the decision-maker. The 
Delphi method is a structured means of incorporating expert 
opinions (usually subjective) through questionnaires and controlled 
feedback to estimate a technology impact and the confidence of 
achieving that impact.3" 


It is also true in the case of US programs that, as in the Kirby and Mavris 
report, that TRL data is often derived subjectively, relying on the subject matter 
expert opinion of program managers, using a method like the Delphi method. 

3. Accuracy OF T-50 TRL Assessment 

Based on the report, “Analytic evaluation on technology acquisition of T-50 
program” published by the Seoul University Aerodynamic Institute, the T-50 
program has 13 critical technology parts. This report lists these critical 
technology parts, and it provides some conclusions. 


e Eight parts (Aerodynamic, Structure, Detail System, Test and 
Evaluation, Cockpit, Design of Shape, Summary of System, 
Training System) have made fairly good progress and could 
achieve the expected goal until the time of weapon research and 
development 


° Four parts (Avionics, Prototype Manufacturing, Flight control, 
Thrust system) have not enough technology and especially avionic 
parts need continuing effort because it has a little chance of 
achieving the goal. Even though flight control part does not make 
good progress, the ongoing effort will achieve the goal. 


e Supply and support system does not have enough technology 
because of a late start. 


31 Forecasting Technology Uncertainty in Preliminary Aircraft Design, Michelle R. Kirby and 
Dimitri N. Mavris, p. 4. 
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Achieved 
Level 
Aerodynamic : 88% 
Structure ; 83% 
Flight Control : 54% 
Avionics : 53% 
Detail System 17% 
Prototype Manufacturing 79% 
Test and Evaluation 77% 
Cockpit 67% 
Thrust System 67% 
Design of Shape 96% 
Summary of System 84% 
Supply and Support 63% 
System 
Training System 74% 


Summary : 70% 


Part Significance(%) 
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Table 7. T-50 TRL Assessment 


4. US Program TRL Assessment 


e B-2 RMP: the program reported having two critical technologies, 
but a formal technology readiness assessment conducted in 
February 2004 concluded that two additional technologies should 
be considered critical. The additional two technologies, the 
receiver/exciter for the electronic driver cards and aspects of the 
antenna designed to help keep the B-2’s radar signature low, are 
not considered fully mature but are approaching maturity.2 


e C-130 AMP: Five of the C-130 AMP's six critical technologies are 
fully mature, as the program is primarily utilizing proven commercial 
and modified off-the-shelf technology for all AMP capabilities. The 
remaining critical technology, the Terrain Following and Terrain 
Avoidance (TF/TA) capability, was demonstrated through the Air 
Force Research Lab’s Quiet Knight advanced technology 
demonstration. 33. The program manager estimate that the 
technologies that make up the Improved TF/TA System were 
assessed as having a TRL of 6 at the time they were transitioned 
from AFRL. 


32 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 32. 
33 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 34. 
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e C-5AMP: we did not assess the C-5 AMP's critical technologies 
because the program used commercial technologies that are 
considered mature. Program officials stated that those technologies 
are in used on other aircraft.34 


e C-5 RERP: The program manager estimate the TRL of 11 critical 
technologies (Auxiliary Power Unit (APU), Engine hardware, Engine 
software, Electrical Power System, Elevator Variable Feel Unit, 
Flight Controls System, Hydraulic Suction Boost Pump, Integrated 
Diagnostic System, Nacelle/Thrust Reverser, Pylon, Wing-to-Pylon 
Attachment Fitting) of the program. 


e F-22: The F/A-22 did not have mature technology at the start of the 
acquisition program. The program included new low-observable 
(stealth) materials, integrated avionics, and propulsion technology 
that were not mature at this time. The Air Force did not complete an 
evaluation of stealth technology on a full-scale model of the aircraft 
until several years into development. It was not until September 
2000, or 9 years into development, that the integrated avionics 
reached a maturity level acceptable to begin product development. 
During development, the integrated avionics was a source of 
schedule delays and cost growth.3° 


° Global Hawk: The program manager estimate the TRL of 11 critical 
technologies (Air Vehicle & Engine, Communications/Information 
Exchange, Enhanced Electro-optical/Infrared Sensors, Enhanced 
Synthetic Aperture Radar, Global Air Traffic Management, Lithium 
Batteries, Multiple Platform - Radar Technology Insertion Program, 
Open System Architecture, Signal Intelligence Phase 1, 2, and 3) 


e JSF: GAO/NSIAD -00-74 “joint strike fighter acquisition” report the 
eight key technology levels at the start of development. 


e JUCAS: The program manager estimate the TRL of six critical 
technologies(Signature Reduction Technologies, Advanced Tactical 
Targeting Technologies, Secure Robust Communications, Force 
Integration, Interoperability, and Global Information Grid (GIG) 
Compatibility, Adaptive Autonomous Operations, Operations in 
Carrier Controlled Airspace) 


° Predator: Three of the predator B's four critical technologies, the 
synthetic aperture radar, the multispectral targeting system, and the 
air vehicle, are fully mature.%6 





34 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 36. 
35 GAO-03-645T, Best Practice, p. 11. 
36 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 102. 
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e V-22 : The program manager estimated that the results of testing 
the MV-22 Block A under the expected range of environmental 
conditions in which it is expected to operate was successfully 
completed. Assessment of whether it will meet its operational 
requirements was made. Problems were identified and corrective 
plans were developed and are being implemented to insure a 
successful Operational Evaluation 


5. Analysis of the TRL 

Our best estimate of TRL at the program development stage is based 
upon input from the program manager. We have some data that indicates TRL 
growth form stage-to-stage. From these data, we have observed the following 

e The growth rate from development to design review stage is 27%. 

e The growth rate the one from design to development stage is 12%. 

These growth rates imply that in order to achieve the GAO benchmark of 
TRL 7 at the production decision stage, it is sufficient to achieve a TRL of at least 
5 at the development phase. Based on this fact, we provide the following 
forecasts: 


e JUCAS, JSF, and F-22 programs may not achieve a desirable TRL 
level at the production stage. 


e T-50, program having TRL 6.3 at the development stage, will 
achieve the goal of TRL 7. 


6. Summary 

GAO and program manager assessed the TRL on 10 US major weapon 
acquisition programs. Each of these 10 major weapon acquisition programs has 
some similarity to the T-50 program. 


GAO has stated that a best practice is to have a TRL of 7, and that this 
level indicates that the program will progress over time without potential cost and 
schedule growth. 


We observed the following facts about JSF and F-22 from our data set 


° Each had an immature technology at the development phase. 

e Program acquisition unit costs were 116.5% on the F-22, and 
26.2% on the JSF, These estimates were made in GAO-05-301. 

° Even though GAO recommends, as a best practice, the use of the 


TRL approach as part of the program management for a major 
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weapons system acquisition, some programs proceeded without 
achieving these levels or doing this analysis. That is, these 
programs started without considering the TRL of the key 
technologies, and therefore failed to lay the basis for reducing cost 
increases and schedule delays. 


On the other hand, DoD does not always agree with the GAO reports, 
GAO 05-271 and GAO 05-301. An example of this is the assessment of JSF 
technology maturity. Furthermore, US programs do not always follow the GAO 
guidance. Development decisions for three programs, F-22, JSF, and JUCAS 
were made without achieving the sufficient TRLs of the key technologies. 


In a comparison between the US programs and the T-50 program, we see 
that the T-50 also started without fully matured TRL. Of course, TRL can be 
developed with the same step of the technical development in science. Some 
weapon development program needs their specific technology part, which have 
little Chance of being developed in science. With the anticipation of the TRL 
growth rate, we could estimate that the T-50 program starting with TRL 6.3 is 
going to achieve the GAO benchmark of TRL 7. 

B. DESIGN DRAWINGS 

1. T-50 Design Release Status 

As integral parts of decreasing the risk of cost increases prior to 
production, the T-50 program implemented two processes: Concurrent 
Engineering (CE) and Computer Aided Design (CAD) 


e Computer Aided Design (CAD): T-50 is designed 100% using 
CAD software. Specifically, T-50 was designed using Computer 
Aided Three-dimensional Interactive Application (CATIA), which is 
often used as a design tool in the aerospace industries, and plays a 
major role in making design drawings, and which had to be made 
prior to the manufacturing. “Although CATIA is a robust design tool, 
it did not satisfy all LMTAS integrated product development (IPD) 
needs. Lockheed Martin Tactical Aircraft Systems (LMTAS) 
integrates several design tools to serve company-specific needs. 
For example, LMTAS selected Computer-Aided Three-Dimensional 
Interface Applications (CATIA) (a solid modeling package) as the 
core design software. LMTAS has identified several specific design 
functions not available with commercial software and has chosen to 
customize or build software to serve these specific needs. 
Computer Mock-up (COMOK) provides an example of enhancing 
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commercially available CATIA software to satisfy LMTAS specific 
needs. The dynamic sharing of design iterations enables 
concurrent, IPD. The COMOK database management system is the 
technology that provides centralized access to multiple 
configurations defined geometrically by CATIA solid models. As 
new programs are initiated, COMOK is used in place of metal 
mock-ups. Since it is available at the start of the design process, it 
improves design integration and quality.”3”. 


e Concurrent Engineering (CE) :In addition to the use of CATIA, T- 
50 used the concept of Concurrent Engineering that is “a business 
strategy which replaces the traditional product development 
process with one in which tasks are done in parallel and there is an 
early consideration for every aspect of a product's development 
process. This strategy focuses on the optimization and distribution 
of a firm's resources in the design and development process to 
ensure effective and efficient product development process.”38 


For the T-50 program, Table 8 provides design release status while Figure 
8 shows the percentage of drawings released over time while using the CATIA 


and CE processes on the process of engineering drawings. 



























































Table 8. T-50 Design Drawing Release 





37 Best Manufacturing Practices, 
http://www.bmpcoe.org/bestpractices/internal/Imtas/Imtas_3.html (accessed November 29, 2005) 


38 Concurrent Engineering, http://best.me.berkeley.edu/~pps/pps/concurrent.html#what 
(accessed November 29, 2005) 
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Figure 8. Percentage of Drawings Released 


2. US Program Design Release Status 


e B-2 RMP: The program has completed and released 71 percent of 
its engineering drawings to manufacturing. The program office has 
scheduled the design readiness review for June 2005 and plans to 
have 85 to 95 percent of its drawings released by that time.°9 


e C-130 AMP: The program office has made progress toward 
meeting its goal of releasing 90 percent of the design drawings by 
design readiness review, scheduled for August 2005. This will be 
nine months sooner than anticipated last year, due to the 
acceleration of key program dates to meet Special Operations 
Command requirements. Currently, 48 percent of the design 
drawings are complete and could be released to manufacturing.4° 


e C-5AMP: The design appears stable as the contractor has released 
100 percent of the drawings for the AMP.41 


e C-5 RERP: The program manager estimate C-5 RERP’s design is 
stable. 3,250 out of total 3,300 were completed before the 
December 2003 at Critical Design Review. 


° F-22 : The F/A-22 design is essentially complete, but it matured 
slowly, taking over three years beyond the critical design review to 
meet best practice standards. The late drawing release contributed 





39 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 32. 
40 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 34. 
41 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 36. 
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to parts shortages, work performed out of sequence, delayed flight 
testing and increased costs. Design changes resulted from flight 
and structural tests.42 


° Global Hawk: The program manager defined that 1,650 out of total 
1,800 were completed at the April 2004 Critical Design Review. 


° JSF: The program estimates 35 percent of the engineering drawing 
packages are expected to be released at the critical design review. 
Also, prototype testing will not be done prior to the design review. 
The design will not be stable until after production begins.4 


e JUCAS: The program manager defined that 2,400 out of total 2,800 
were completed at the September 2005 Critical Design Review. 


e Predator: The program office expects 94 percent of the expected 
increment one drawings to be completed by the April 2006 critical 
design review, which has been delayed about seven months. 
Program officials acknowledge that additional drawings will be 
needed for subsequent increments.44 


e V-22: The program manager defined that 7,230 out of total 7,490 
were completed at the September 2002 Critical Design Review. 


Figure 9 shows the percentage of releasable design drawings on the US 
programs based on the GAO report and input from program managers, before 
CDR. 





42 GAO-02-701, Best Practices, p. 25. 
43 GAO-05-271, Joint Strike Fighter Acquisition, p. 17. 
44 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 101. 
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Figure 9. Releasable Drawings before CDR of US programs 


3. Proportion of Drawings Released at CDR 

To examine design stability, we reviewed the time schedule for 
transitioning the engineering drawings from the engineering side of the house to 
the manufacturing side of the house. There is a natural need to demonstrate, 
before beginning initial manufacturing, that the design of the product meets the 
stated requirements of the weapons acquisition program. To determine the 
design stability, and to judge whether the product is ready for the next phase of 
development, CDR begins to review the product from the perspective of the 
product’s component, then continues its review at the subsystem level, and, 


finally, at the integrated, system level. 


GAO’s best practice benchmark for design stability requires at least 90% 
of the engineering drawings to be completed prior to Critical Design Review 
(CDR). The time schedule for giving the engineering drawing from engineering to 
manufacturing is used to examine design stability, and we developed and 
analyzed, as a measure of effectiveness, the percent achievement of the number 


of design drawings which are releasable to the manufacturing of each program. 
39 


The data of C-5RERP, Global Hawk, JUCAS(X-47), V-22 and T-50 were 
estimated and provided by each program manager. On the other hand, data for 
the other programs are based on the GAO report, which had already collected 
and judged the achievement. The GAO report also shows the achievement 
before CDR of the B-2RMP and C-130AMP programs. The Global Hawk 
program manager also provided the specific drawing data at the start of the 
program. The column of the expected value represents the expected 


achievement from the program start to the CDR. 


The table below displays the data we have developed on engineering 


drawings. 


Achievement Releasable : Criteria 
before CDR(%) |drawings at CDR(%)| FxPectation | (go) 
B-2 RMP 0.71 0.90 0.19 0.9 
C-130 AMP 0.48 0.90 0.42 0.9 
C-5 AMP - 1.00 - 0.9 
C-5 RERP - 0.98 0.9 


Aircraft 




















F-22 - | 0.26 | 0.9 


Global Hawk 0.92 0.9 
JSF 0.35 0.9 
JUCAS 0.86 0.9 
Predator 0.94 0.9 
V-22 0.97 0.9 
T-50 0.67 0.9 






































Table 9. Design Drawing Release Status 


The following Figures 10 and 11 illustrate that the Matured Key technologies 
make the design stability keep going well during the weapon acquisition process. 
The seven programs, B-2 RMP, C-130 AMP, C-5 AMP, C-5 RERP, Global Hawk, 
Predator, and V-22, which have at least five at the development phase, have a 
satisfactory level of design drawings at CDR. Even if six critical technologies of 
the JUCAS were not matured at the development phase, the achievement of 


design drawing is ready to enter the stability level. The releasable drawings of T- 
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50 program at CDR is not at a sufficient level to assure design stability, even 
though it starts with TRL above 5. Two programs having the lower TRL 
achievement at development phase also show the lower achievement of design 


stability. 
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Figure 10. Design Drawing Release versus Criteria 
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Figure 11. CDR Drawings / Total Drawings 
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The following figure shows the expectation of the design drawings from 
development to the time of CDR. As we see, the data of only three programs, B- 
2RMP, C-130AMP and Global Hawk, are available to represent, because each 
program is not managed and monitored by each program manager according to 


the time phase. 
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Figure 12. Achievement versus Expectation 


A. R&D Cost versus Number of Drawings 

The R&D cost for five programs except T-50 is on the basis of GAO report 
and R&D cost of the T-50 is from the published book for the advertisement of T- 
50. We can obtain the ratio of the number of drawings from the Critical Design 
Review (CDR) for all programs which we have analyzed, but obtaining the exact 
data on the number of drawings is hard to get even from the program managers. 
The number of drawings of each program in the table below comes from the 
program managers. To judge the relation between the R&D cost and number of 
drawings, we divided the R&D cost by the number of drawings and tried to find 
some results. All R&D costs are normalized to the FY2005. A five percent 
inflation rate is used to normalize the T-50 weapon development program. The 
table below display the R&D cost versus the number of drawings. 
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Program R&D cost # of drawings R&D/# of drawings 


C-5 RERP_| $1,537.4 3300 0.47 
Global Hawk | $2,528.9 1800 1.40 














V-22 $10,723.7 7490 1.43 
T-50 $2,790 12290 0.20 
JUCAS $4,042.0 2800 1.44 

= ( FY05 dollars in millions) 




















Table 10. R&D Cost Data 


It is analytically pleasing when three numbers are almost the same, as is 
the case for Global Hawk, JUCAS and V-22. It seems that $1.4 M is some sort of 
constant for new aircraft. We do not know what causes this constancy but 
perhaps it can be a topic for further research. T-50, which does not fit the pattern, 
is a tactical aircraft, whereas Global hawk and JUCAS are Unmanned Air 
Vehicle’s (UAV) 
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Figure 13. R&D Cost/# of Drawings versus # of Drawings 
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5. 
The figure below shows the regression line on the percent achievement of 


Regression on the Engineering Drawings 


releasable design drawings at CDR versus R&D cost. 


Unmanned programs, Predator, Global hawk and Jucas, have a 
linear relationship along the R&D cost. These programs are 
identified with circles in Figure 14. 


Advancing program, C-5AMP, C-5RERP, C-130AMP and B-2RMP, 
have a big variation of achievement of releasable design drawings 
depending on each program. These programs are identified with a 
square in Figure 14. 


V-22 also arrived high level of drawing achievement as other 
advancing programs even though it has higher R&D costs than 
other advancing programs. 


The new and advanced program, F-22 and JSF with substantial 
R&D costs arrived a lower level of CDR achievement. These 
programs are identified with triangles in Figure 14. 


T-50 program, which is a new program, has better level than F-22 
and JSF, but a low achievement when we considered the low R&D 
cost programs. 
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Figure 14. Percentage Achievement at CDR versus R&D Cost 


Figure 13 represents the percent achievement of design drawings at CDR 


versus number of design drawings of the five programs. Our efforts to find the 


exact number of design drawings had trouble in judging whether the program 
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manager had updated the number of each program during the whole 
development period. Therefore, we can obtain only reliable number of design 
drawings on the five programs, Global Hawk, JUCAS, C-5RERP, v-22 and T-50. 


e The only regression analysis on the manned programs, C-5RERP, 
V-22 and T-50 result in a difficulty of finding the relationship 
between percent achievement and number of drawings due to the 
little data information. 


e This regression trend shows that the uncomplicated programs (that 
is, those having fewer design drawings) arrived at a more 
satisfactory level of releasable design drawings than the programs 
having more design drawings. 


e T-50 has more design drawings than advanced program and 
unmanned program. Its achievement of design drawing, 67 percent, 
at CDR is below the regression line. If the T-50 also achieved the 
about 90 percent of drawings at CDR, the regression line would be 
the flat line and will have very little slope of regression line. 
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Figure 15. Percentage Achievement at CDR versus # Drawings 


6. Summary 

The US GAO had set the benchmark that the achievement of engineering 
drawing stability requires at least 90 percent of engineering drawings to be 
releasable by CDR. We observed the following from our dataset: 
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Even thought this benchmark gives good guidance by which to 
judge the design stability of a major weapon acquisition program, 
some programs do not follow the GAO guidance 


The seven programs, B-2 RMP, C-130 AMP, C-5 AMP, C-5 RERP, 
Global Hawk, Predator, and V-22, which have achieved a TRL 
score of at least 5 at the development phase, also meet or exceed 
GAO’s 90 percent engineering drawing benchmark. 


Even if six critical technologies of the JUCAS are not matured at 
the development phase, the program has achieved the design 
drawing benchmark and is stable on this dimension. 


Two programs, F-22 and JSF, having the lower TRL achievement 
at development phase also show the lower achievement of design 
stability. 


The percent of achievement is correlated with program complexity, 
where complexity is either research and development cost or 
number of design drawings. 


The relationship between the achievement of releasable design drawings 


and level of R&D cost shows the following: 


Unmanned programs, Predator, Global hawk and JUCAS, have a 
linear relationship along the R&D cost. 


Advancing program, C-5AMP, C-5RERP, C-130AMP and B-2RMP, 
have a big variation of achievement of releasable design drawings 
depending on each program 


V-22 also arrived high level of drawing achievement as other 
advancing programs even though it has high R&D cost than other 
advancing programs. 


The new and advanced program, F-22 and JSF, having a very high 
R&D achieved a lower level of CDR achievement. 


We investigated the relationship between the achievement of releasable 


design drawings and the number of design drawings for the five programs for 


which we have data. The observations are: 


The only regression analysis on the three data points which 
represent manned programs, C-5RERP, V-22 and T-50, has poor 
statistics and we concluded that it is difficult to find a linear 
relationship between percent achievement and number of drawings. 
This is probably due to the little data information available to us. 
This regression shows that the less complicated programs having 


46 


fewer design drawings achieved a satisfactory level of releasable 
design drawings than the programs having many number of design 
drawings. 


We considered the T-50 program with the following observations: 
° T-50 TRLs are above 5 


e T-50 achived releasable design drawings of 67 percent at CDR, 
which is below GAO guidance, so that T-50 is at risk for not 
showing design stability. 


T-50 has more design drawings than advanced program and unmanned 
program among five programs 
C. PRODUCTION 
1. T-50 SPC Process Plan 
This paragraph describes the T-50 SPC plan during the Full Scale 
Development and Production period and is translated from the T-50 SPC plan. 
a. Definition of SPC of T-50 Program 
Statistical Process Control (SPC) is a prevention focused process 
management method that is used to identify sources of malfunctions that are 
occurred by inherent variances, materials, workers, management methods, 
measuring methods and working environment, through quality characteristic 
analyses which separate, measure and quantify causes of malfunctions through 
statistical methods. 
b. Measuring, Analyzing and Improving Process 
SPC plans monitoring, measuring, analyzing and continuous 


improvement process and execute them for relevant events as follows: 


° Verification of product suitability 
e Ensuring suitability of quality management systems, and 
° Continuous improvements of effectiveness of quality management 


systems. Monitoring, measuring, analyzing and _ continuous 
improvement process also include decision making process on 
methods and a level of implement of SPC. 
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Cc. Statistical Method 
By implementing statistical methods, we can ensure continuous 
and stable quality of products. This is achieved through forecasting potential 


problems and then devising countermeasures to prevent these forecasted 


problems. 

e Analyze collected data through statistical method and manage 
factors that affect product quality. 

e A whole inspection of product is a general rule. However, when 
implementing a sample inspection, it is require to establish a 
sample inspection plan and get the plan approved by the 
government. 

e During prototype production, SPC is implemented as a data 
collecting stage. Upon full scale production, KC’s will be identified 
for fully implementing SPC. 

2: T-50 Statistical Process Control 


We reviewed information from the T-50 program, which is at the beginning 
of the full-scale production phase. T-50 prototype production was performed 
without fully developed statistical control. Below is information about the T-50 
Statistical Process Control (SPC). 


e SPC (Statistical Process Control), during the stage of Full Scale 
Development (FSD), was defined as a phase of data collection that 
is necessary to identify Key Characteristics (KC’s) for the mass 
production. 


e SPC should be fully implemented upon identifying KC’s during 
mass production. However, according to one of our assembly 
engineers, there has not been an instance where SPC had been 
implemented directly for each component of T-50 model. 


e However, there have been a number of cases where SPC had 
been implemented to improve the production process electronically. 
(i.e., reducing ‘lead time’ through process improvement) 


3. US Program Production Knowledge 
The third dimension of GAO’s knowledge-based approach to evaluating 


weapons-systems acquisition is Statistical Process Quality Control (SPQC). We 
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examined each aspect of our programs and found, surprisingly, that most 


programs are not doing this. The following table45 displays the results. 





Programs 


Description 





B-2 RMP 


“Production maturity metrics are planned to be formulated during 
development. These metrics, which may or may not include 
manufacturing process control data, are planned to be used as 
measures of progress toward production maturity during a 
production readiness review prior to the start of production in 
February 2007”. 





C-130 AMP 


“Funding reductions in fiscal years 2003 and 2004 delayed the C- 
130 AMP’s development program, which resulted in a 
rescheduling of program milestones and rebase lining of the 
program. The design review, low-rate initial production, and 
production readiness decisions were all delayed. While program 
Officials stated that the delay in schedule would provide more time 
to resolve issues with the TF/TA technology and software, the 
delay in fielding was not acceptable to the Special Operations’. 





C-5 AMP 


“We could not assess the production maturity because most 
components are readily available as commercial-off-the-shelf 
items. This equipment is being used on other military and 
commercial aircraft. In addition, the C-5 AMP is incorporating 
many other off-the-shelf systems and equipment, such as the 
embedded global positioning system, the inertial navigation 
system, and the multifunction control and display units. To ensure 
production maturity, the program office is collecting data regarding 
modification kit availability and the installation schedules.” 





C-5 RERP 


“We did not assess the C-5 RERP’s production maturity because 
the Air Force is buying commercially available items. However, we 
expect that production maturity would be at a high level because 
the engines have been commercially available for many years.” 








F-22 





“The program office stopped collecting process control information 
in November 2000. The contractor estimated that nearly half of the 
key processes had reached a marginal level of control, but not up 
to best practice standards. The Air Force relies on the contractor's 
quality system to verify manufacturing and performance 
requirements are being met. However, the Air Force has not 
demonstrated the F/A-22 can achieve its reliability goal of three 
hours mean time between maintenance. It does not expect to 
achieve this goal until 2008 when most of the aircraft will have 





45 GAO 05-301, Assessments of Selected Major Weapon Programs, pp. 32, 34, 36, 38, 64, 


68, 80, 102, 118. 
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Programs 


Description 





already been bought. Best practices call for meeting reliability 
requirements before entering production. As of the current 
production estimate by approximately mid-October 2004, the Air 
Force had only demonstrated about 22 percent of the reliability 
required.” 





Global 
Hawk 


“Although production experience and lessons learned on the RQ- 
4A will benefit the RQ-4B program, the new model requires 
different and more complex manufacturing processes and tooling 
than the original model. Officials have not implemented, and do 
not plan to implement, a comprehensive statistical process control 
program to demonstrate that new manufacturing processes are in 
control and capable of meeting cost, schedule, and quality targets. 
Officials have started to identify critical manufacturing processes 
and will continue to collect performance data such as defect and 
rework rates to measure product quality. There are continuing 
concerns about the quality and timeliness of several key 
subcontractors, which negatively affect cost and schedule of both 
design and production work.” 





JSF 


“The program office is collecting information on the JSF production 
processes. The contractor is currently in the process of identifying 
the key characteristics, critical manufacturing processes and 
capturing some early data. At the time of the production decision, 
the program will not have demonstrated that the aircraft can be 
produced efficiently or with expected reliability.” 





JUCAS 


“The J-UCAS program began in October 2003 with technologies 
that officials project will sufficiently mature to support a possible 
2010 start of operational system development, so there is no data 
for production.” 





Predator 


“Program officials said the contractor does not plan to use 
statistical process controls to ensure product quality. Instead, they 
plan to use other quality control measures such as scrap, rework, 
and repair to track product quality. Also, initial operational testing 
of increment one, which is to demonstrate a product is ready for 
production, is not scheduled to be complete until September 2007. 
Testing for remaining increments has not been determined.” 








V-22 





“Process management is becoming more robust at the final 
assembly site on each major fixture assembly using Six Sigma. 
Program Officials point to the delivery of aircraft as an indication of 
manufacturing maturity.” 





Table 11. 


US Program Production Knowledge (After: GAO 05-301) 


50 





4. Summary 

The GAO benchmark suggests a standard rule for describing the 
production maturity and for estimating if the programs are under control during 
the weapon production phase. As the GAO states it: 

To determine if a product’s design is reliable and producible, 

successful programs use statistical process control to bring 

manufacturing processes under control so they are repeatable, 

sustainable, and consistently producing parts within quality 

standards. The collection of process control data prior to a 

production decision can enable a smooth transition from product 

development to the production phase.46 

One of the frequently heard comments about the difficulty of implementing 
SPC is the high level of training required of personnel and the high costs of these 
personnel. Due to the typical shortage and high cost of the personnel needed to 
perform such tests, we focus only on the critical processes necessary to meet the 
reliability and performance standards. GAO used the Process Capability Index, 
which can be used to compute a product defect rate that permits us to assess 
such critical process and the index is measured the application of statistical 


control. 


When we focus on the critical process, the SPC data needed to judge a 
program’s production maturity depends on the particular program manager. 
Based on the program manager data and some reports, most programs in our 
dataset do not follow the GAO’s SPQC guidance. As the GAO states: “While 
the absence of the data does not mean that production processes were immature, 
it does prevent an assessment against an objective standard.”4” On the other 
hand, the production decision of some programs will be made later, and some 


programs have a plan to follow the SPC. 


Our conclusions on the programs in our dataset follow. 


e C-130AMP, JUCAS, V-22: the programs are under development, 
so it is too early to judge the SPC. 


46 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 12. 
47 GAO-05-301, Assessments of Selected Major Weapon Programs, p. 12. 
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C-5 AMP, C-5 RERP: production maturity seems assured since 
many aspects of this program are readily available as commercial- 
off-the-shelf items. 


B-2 RMP: SPC are planned to be formulated later, as the programs 
enter their production phase. 


F-22, Global Hawk, JSF, Predator: we can judge that the programs 
do not follow the GAO guidance 


T-50: prototype production was performed without fully developed 
statistical control and the program has just entered the full-scale 
production phase. While SPC should be fully implemented upon 
identifying Key Characteristic(KC’s) during mass production, our 
information from assembly engineers is that there has not been an 
instance where SPC has been implemented directly for any 
component of T-50 prototype model. On the other hand, there have 
been a number of cases where SPC had been implemented to 
improve the production time. Looking further into the future, T-50 
program has a specific plan to implement SPC during the Full Scale 
Development and Production period. 
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IV. CONCLUSION 


A. SUMMARY 
Technology development is the important factor in estimating whether a 


weapons-development program could meet requirements without schedule 


delays and cost overruns. Acquisition-program management also relies on the 


knowledge-based approach in developing new weapon programs and it ensures 


achievement of key results through high levels of knowledge. To support 


program management, the US GAO has developed a methodology, which 


implements three knowledge points. 


Technology maturity 


GAO attempts to find the key technologies, which are measured on 
a scale of nine levels of technological maturity. Starting a product 
development with a technology readiness level of seven is a GAO- 
identified “best practice”, and it appears to ensure the satisfactory 
progress of weapon acquisition programs. 


Design stability 


Successful programs obtain design stability at the time of critical 
design review (CDR), and CDR results serve as criteria for 
estimating the stability of engineering drawings released. The 
benchmark that the US GAO has set is that engineering-drawing 
stability is achieved if it is true that at least 90 percent of 
engineering drawings are completed (or at least releasable) before 
the CDR. 


Controlled production stability 


Products manufactured with statistical process controls in place are 
better able to govern costs and to stay within schedule than 
programs that do not have SPC in place. To evaluate such 
maturity, the components of statistical process control are used and 
critical manufacturing processes are monitored for statistical data. 
To measure the performance of the process, a process-capability 
index (CPK) is used to determine how a process is running 
compared to its specification limits and to measure production- 
process capability within specified limits. The CPK index below 1.33 
indicates that the manufacturing process lacks statistical control 
and acceptable consistency. 


To monitor and evaluate the programs’ maturity and stability, we collected 
data from some US programs, B-2 RMP, C-130 AMP, C-5 AMP, C-5 RERP, F-22, 
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Global Hawk, JSF, JUCAS, Predator, and V-22, relevant to the aircraft 
acquisition and advancing program to determine if the acquisition also follow the 
GAO guidance or not. These data are also compared to the T-50 acquisition 
program data to estimate if the program would make good progress in the 


research and development phase. 


Interestingly, even though the US GAO had been trying to enhance the 
military capability and readiness, some acquisition program did not follow the 
guidance of the GAO. The following table shows the status of whether each 
program programs in our dataset follows the GAO guidance or not. In this table, 
we followed the following rules: 

e A TRL score above 7 is considered to follow the GAO guidance 


e 90 percent achievement of engineering drawings at CDR was 
considered to follow the GAO guidance. 


e The production followed by the SPC is considered to follow the 
GAO guidance no matter what the CPK is. If some program is 
under development, we assume that SPQC is not available. 


Programs Engineering Drawings 








B-2 RMP Yes 





C-130 AMP Yes 





C-5 AMP Yes 





C-5 RERP Yes 





F-22 No 





Global Hawk 





JSF 





JUCAS 





Predator 





V-22 





T-50 














Table 12. Observance of GAO Benchmarks 
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This objective of this thesis is to investigate if the GAO knowledge based- 


methodology could be used in the T-50 program and, if possible, to judge the 


progress on the base of GAO methodology with the collected data. We can find 


some conclusions as follows. 


We can find the technical readiness level, releasable design 
drawings at CDR and Statistical Process Control plan on the T-50 
program, so we can judge the T-50 program on the basis of GAO 
report. 


Since the T-50 program achieved a TRL of 6.3 at the development 
stage, we believe that the T-50 program will achieve the goal of 
TRL 7. 


The releasable drawings of T-50 program at CDR is not a sufficient 
level to ensure design stability, even though it started with a TRL 
above 5. T-50 has more design drawings than the advanced 
programs in our dataset as well as more design drawings than the 
unmanned programs in our dataset. The T-50 program’s 
achievement of design drawing, 67 percent, at CDR is below the 
“best practice” guidance of the GAO. 


T-50 prototype production had been performed without fully 
developed statistical control and production is on the beginning 
phase of mass production. However, there has not been an 
instance where SPC has been implemented directly for each 
component of T-50 prototype model. Fortunately, T-50 program 
have a specific plan of SPC during the Full Scale Development and 
Production period. 


B. RECOMMENDATIONS FOR FUTURE RESEARCH 
During this research, we identified some further research areas. 


The first is to forecast the maturation of technology over time on the basis 


of Kirby and Mavris report describing the impact of emerging technology on the 


probabilistic technology maturation process. Even though this report focuses on 


the estimation of low level of technology maturity, if we can obtain the reliable 


TRL of weapon program, which is developed with TRL above 5 from the 


development start to production, we can compare the TRL improvement with 


estimation of the report. 


The second is to find the relationship between the number of drawings 


and research and development cost. We note that the higher number of 
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designing drawings, the higher research and development cost for the programs 
in our dataset. This topic needs additional data to explore and identify the 


statistical relationship between two items. 


The third is to find the statistical process control data for the entire 
analysis of the GAO methodology. The environments at which most program 
managers do not try to follow the GAO guidance make it hard to estimate product 
stability. Further research should explore the SPC and CPK data. It is possible 
that full-scale analysis of GAO methodology would also yield good guidance to 


the new developing weapon acquisition program. 
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